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(54) Systems and methods for providing access to multimedia messages for remote users using 
telephones and facsimile machines 



(57) A system for receiving, sending and creating 
multimedia messages includes an audio communica- 
tion device, a visual output device, a remote access sys- 
tem and a multimedia-message system. The audio com- 
munication device is coupled to the remote access sys- 
tem to provide audio input and control signals, and also 
for receiving the audio portions of multimedia messag- 
es. The remote access system also couples the visual 
output device to the multimedia message system for ac- 
cessing the visual portions of multimedia messages. 
The multimedia message system has the capability to 



generate an image of the message in response to a user 
request and send that image to the remote access sys- 
tem, and then to the visual output device to the user. 
The image is augmented to include indices that the user 
may input via the audio communication device to per- 
form specified actions with the various portions of the 
message. This functionality is particularly advanta- 
geous because it extends the usefulness of the multi- 
media message system so that it may be used for full 
access to the systems capabilities with a conventional 
audio communication device such as a telephone. 
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Description 

CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] The present invention relates to U S. Patent 
Application Serial No. 09/407,01 0, entitled "Method and 
Apparatus for Generating Visual Representations for 
Audio Documents," filed on September 28, 1 999. which 
is incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present invention relates to systems and 
methods for generating, storing and retrieving multime- 
dia messages that may include audio, text documents, 
images, web pages (URLs) and video. In particular, the 
present invention relates to a system and method for 
providing remote access to such systems. Still more 
particularly, the present invention relates to a system 
and method that allows remote access to multimedia 
messages using conventional telephones and facsimile 
machines or other conventional display devices. 

2. Description of the Background Art 

[0003] The telephone and voice mail are basic com- 
munication tools whose use has become widespread. 
Audio communication provides the advantage of a rich 
medium in which expression as well as content can be 
conveyed. Voice mail systems in which the user Is able 
to create, store, send, forward, and attach audio mes- 
sages to others are well known, and are part of the com- 
munication tools used by many on a daily basis. One 
problem with such voice mail systems is that they pro- 
vide for messaging only in the form of audio messages. 
Yet another shortcoming of voice mail systems is that 
the user is not given any context or other information for 
a voice message as can be provided with a graphical 
display. The only information such systems typically pro- 
vide is how many messages are in a person's in box. In 
most cases, the user is not able to identify whom the 
messages are from, their duration or whether they are 
a siring of several messages from different speakers. 
Therefore, in those instances when documents, web 
pages (URLs), video or other media are the best way to 
communicate, voice mail systems are unable to receive, 
send and store messages in these media. 
[0004] Another tool that has become an integral part 
of one's daily communication is e-mail. The use and pro- 
liferation of e-mail has become widespread because of 
the speed, cost and ease of use. Initially, e-mail mes- 
sages were confined to being text only. Thus, e-mail pro- 
vided only a form of written communication that was 
much faster than traditional mail. The development of 
e-mail now allows for the attachment of documents, vid- 
eo clips, audio clips, web pages (URLs) and other me- 
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dia. However, a continuing difficulty with e-mail is that' 
these attachments are difficult to access. Moreover, 
they are not presented to the user in any meaningful way 
to convey their content or interrelationship. 
[0005] Even more significantly, e-mail that is able to 
provide such multimedia content can typically only be 
accessed using a system such as a personal computer. 
This makes it very difficult to be able to access e-mail 
when traveling or when a computer is not readily acces- 
sible. There are a variety of mobile devices such as wire- 
less telephones, pagers and palm computing devices 
that attempt to provide remote access to e-mail, howev- 
er, these devices are not able to effectively provide ac- 
cess to and display e-mail attachments. 
[0006] Thus, there remains a need for a system and 
method that is capable of providing remote access to 
rich multimedia messages. Furthermore, there is a need 
for a system and method that allows the creation of mul- 
timedia messages remotely. 

SUMMARY OF THE INVENTION 

[0007] The present invention overcomes the deficien- 
cies and limitations of the prior art by providing a system 
and method for receiving and creating multimedia mes- 
sages. In one embodiment, the system includes an au- 
dio communication device, a visual output device, a re- 
mote access system and a multimedia message sys- 
tem. The audio communication device is coupled to the 
remote access system for bi-directional communication 
with the multimedia message system. The audio com- 
munication device is used to provide audio input and 
control signals, and also for receiving the audio portions 
of multimedia messages. The remote access system al- 
so couples the visual output device to the multimedia 
message system. The user receives visual portions of 
multimedia messages from the multimedia message 
system via the visual output device. The remote access 
system is coupled or may be part of the multimedia mes- 
sage system. The multimedia message system is capa- 
ble of creating, receiving, storing and providing multime- 
dia messages that may include audio, text documents, 
images, web pages (URLs) and video. 
[0008] In one aspect of the invention, the multimedia 
message system has the capability to generate an im- 
age of the message in response to a user request and 
send that image to the remote access system, and then 
through the visual output device to the user. Further- 
more, the image is augmented to include indices that 
the user may input via the audio communication device 
to perform specified actions with the various portions of 
the message. This functionality is particularly advanta- 
geous because it extends the usefulness of the multi- 
media message system so that it may be used for full 
access to the. system's capabilities with a conventional 
audio communication device such as a telephone. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



[0009] The present invention will be best be under- 
stood with reference to the following drawing in which: 
[0010] Figure 1 is a block diagram of a first embodi- 
ment of a multimedia message system constructed in 
accordance with the present invention. 
[001 1] Figure 2 is a block diagram of a second, alter- 
nate embodiment of a multimedia message system con- 
structed in accordance with the present invention. 
[0012] Figure 3 is a block diagram of a preferred em- 
bodiment of a remote access system for a multimedia 
message system constructed in accordance with the 
present invention. 

[0013] Figures 4A-4E are block diagrams of various 
exemplary embodiments of possible configurations of 
the remote access system for a multimedia message 
system using various conventional devices as the audio 
communication device and the visual output device. 
[0014] Figure 5 is a block diagram of a third and pre- 
ferred embodiment of the multimedia message system 
constructed in accordance with the present invention. 
[0015] Figure 6 A is a block diagram of a preferred em- 
bodiment of a multimedia message system including re- 
mote access capability in accordance with the present 
invention. 

[0016] Figure 6B is a block diagram of a preferred em- 
bodiment for a memory of the multimedia message sys- 
tem of Figure 6A. 

[0017] Figure 7 illustrates a graphical representation 
of a multimedia message according to the present in- 
vention. 

[0018] Figure 8 is a flow diagram of a top-level proc- 
ess for retrieving and responding to a multimedia mes- 
sage in accordance with the present invention. 
[001 9] Figures 9A and 9B are a flow diagram of a pre- 
ferred method for retrieving an image of a multimedia 
message in accordance with the present invention. 
[0020] Figure 1 0 Is a flow diagram of a preferred meth- 
od for determining a delivery mechanism for the image 
of the message. 

[0021] Figures 11 A and 11 B are a flow diagram of a 
preferred method for producing or generating a remotely 
usable image of a multimedia message in accordance 
with the present invention. 

[0022] Figures 12A and 128 are a flow diagram of a 
preferred method for retrieving visual or audio portions 
of a multimedia message in accordance with the present 
invention. 

[0023] Figure 13 is a flow diagram of an exemplary 
method for using indices and delivering content of a mul- 
timedia message in specific ways for one embodiment 
of the present invention. 

[0024] Figure 1 4 is a flow diagram of a preferred meth- 
od for remotely composing and replying to a multimedia 
message according to the present invention. 
[0025] Figure 1 5 illustrates a graphical representation 
of a deliverable image or list of documents for remote 



use in responding to a multimedia message. 
[0026] Figure 1 6 is a flow diagram of a preferred meth- 
od for remotely creating and sending a message includ- 
ing a visual object. 
5 [0027] Figure 17 is a flow diagram of a preferred meth- 
od for selecting a visual object for attachment in a mul- 
timedia message. 

[0028] Figure 1 8 is a flow diagram of a preferred meth- 
od for remotely inputting a visual object into the multi- 
10 media messaging system. 

[0029] Figure 1 9 is a flow diagram of a preferred meth- 
od for processing and automatically positioning an input 
object in a message. 

15 DETAILED DESCRIPTION OF THE PREFERRED 



EMBODIMENTS 

[0030] Referring now to Figure i , there is shown a 
block diagram of a first embodiment of a multimedia 
20 message system i ooa constructed in accordance with 
the present invention. The system 100a preferably in- 
cludes an audio communication device 104, a visual 
output device 106, a remote access system 108 and a 
multimedia message system 11 0. A signal line l24cou- 
25 pies the audio communication device 1 04 to the remote 
access system 108 for bi-direction communication. A 
signal line 126 couples an output of the remote access 
system 1 08 to an input of the visual output device 106. 
In this first embodiment, the remote access system 108 
30 is separate from the multimedia message system 110 
and is coupled thereto by bi-direction signal line 128. 
The multimedia message system is a computer system 
for creating, receiving, storing and providing multimedia 
messages that may include audio, text documents, im- 
35 ages and video. An image of an exemplary message 
700 is show in Figure 7 as will be described later. 
[0031] The audio communication device 104 is pref- 
erably a transceiver device that is capable of: 1) receiv- 
ing audio signals and converting them to sounds per- 
40 ceptible by a user 102; 2) receiving sounds, converting 
sounds to audio signal, and sending the audio signal, 3) 
generating control signals using a keypad or other input 
device that is part of the audio communication device 
104. Thus, audio communication device 104 receives 
45 and sends sounds to and from the user 102, and also 
receives and sends audio signals to and from the remote 
access system 108. For example, the audio communi- 
cation device 104 is preferably an ordinary telephone or 
a wireless telephone. In such a case, the signal line 124 
50 is the telephone line that connects the telephone to an- 
other telephone, a PBX or a computer. Thus, the user 
102 is able to receive and send audio signals with a tel- 
ephone, and is also able to send control signals by send- 
ing the touch tones that are output by telephones when 
55 the keys of the telephone are depressed. While the au- 
dio communication device 104 will subsequently be re- 
ferred to throughout this application as a telephone, 
those skilled in the art will recognize that the audio com- 
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munication device 104 may be any device that provides 
the functionality just described. 

[0032] The visual output device 106 is preferably an 
output device capable of generating an image in any vis- 
ual form. The visual output device 106 is coupled to the 
remote access system 1 08 and is responsive to signals 
output by the remote access system 1 08. It is preferably 
the remote access system 1 08 that generates the sig- 
nals and makes sure those signals are in a format ac- 
ceptable to the visual output device 1 06. This is done 
based on the type of visual output device 1 06 with which 
the remote access device 108 is communicating. The 
visual output device 106 is preferably a facsimile ma- 
chine of a conventional type. Such a facsimile machine 
is capable of generating an image or printed page In re- 
sponse to a signal from the remote access system 108. 
Alternate embodiments for the visual output device 1 06 
include: 1 ) a printerconnectedtothe remote access sys- 
tem 1 08 by a network; 2) a display device such as a CRT 
or LED display connected to the remote access system 
108 in a conventional manner; 3) a computer coupled 
to a printer and to the remote access system 108 in a 
standard fashion; and 4) a palm computing device in- 
cluding a display and a wireless transceiver. Such alter- 
nate embodiments will be described below in more de- 
tail with reference to Figures 4A-4E. However, it should 
be understood that much of the functionality offered by 
the present invention can be utilized without having the 
visual output device 106 as part of the system as will be 
described specifically with reference to Figure 4A. 
[0033] The multimedia message system 110 is pref- 
erably a message system for creating, storing, sending 
and retrieving rich multimedia messages. The messag- 
es may include audio, text, video, graphic, etc. The mul- 
timedia message system 110 also includes chronicles 
that refer to a group of messages linked by content. For 
example, a chronicle may include an initial message and 
the replies of several addressees as well as all the por- 
tions of those messages such as text, video, graphic, 
etc. For the present invention, chronicles and messages 
will be considered to be the same. The multimedia mes- 
sage system 1 1 0 allows users to create messages, send 
messages, delete messages, reply to messages, and 
retrieve messages. An exemplary multimedia message 
system n o is disclosed in pending U.S. Patent Applica- 
tion Serial No. 09/407,010, entitled "Method and Appa- 
ratus for Generating Visual Representations for Audio 
Documents." filed on September 28. 1999, which is in- 
corporated herein by reference. In particular, the multi- 
media message system 1 10 is the TouchVerse messag- 
ing system developed by Ricoh Corporation of Santa 
Clara, California. The multimedia message system 110 
is preferably a computer or server that includes pro- 
grams and routines for providing the messaging func- 
tionality described above and in the referenced patent 
application. For example, the server could be similar to 
a web server that stores messages (in contrast to web 
pages) and makes the messages accessible to others. 



The server Is coupled to the remote access system 108 
in a conventional manner such as by a network connec- 
tion, the Internet, a modem, or a dedicated connection. 
The remote access system 108 is coupled or may be 
5 part of the multimedia message system 110 (See Figure 
5). 

[0034] The remote access system 108 provides the 
interface between the multimedia message system 110, 
and the audio communication device 1 04 and the visual 
10 output device 1 06. The remote access system 108 also 
Includes a plurality of units, as will be described in more 
detail below with reference to Figure 3, that enable the 
creation of messages, the retrieval of audio and visual 
portions of messages under the control of the audio 
15 communication device 1 04. The remote access system 
108 preferably has a plurality of input/output ports for 
connecting by the various signal lines 124, 126. 128 to 
these other systems 110 and devices 1 04. 106. For ex- 
ample, the remote access system 108 interfaces with 
20 the audio communication device 1 04 in a manner similar 
to the operation of a voice mail system for the creation 
of audio message portions, the storage of audio mes- 
sage portions, the sending of audio message portions, 
and the retrieval of audio message portions. The keypad 
25 of the audio communication device 1 04 may be used as 
a control input in a manner like that currently available 
with voice mail systems. Moreover, the functionality pro- 
vided in part by the multimedia message system 110, 
and the remote access system 1 08 effectively serves as 
30 a translator to translate the control input into a format 
understandable by the multimedia message system 
110. and for passing audio signals between the multi- 
media message system 110 and the audio communica- 
tion device 104. Similarly, the remote access system 
35 108 preferably interfaces with multimedia message sys- 
tem 1 1 0 in a similar manner such as over a network or 
by direct calls or interrupts to the multimedia message 
system 1 1 0. Finally, the remote access system 1 08 pref- 
erably interfaces with the visual output device 1 06 to use 
40 it as means for providing visual output to the user 1 02. 
This can either be by generating a printed image or by 
providing an image on a screen depending on the form 
that the visual output device 126 lakes. 
[0035] Referring now to Figure 2, a block diagram of 
45 a second, alternate embodiment of a multimedia mes- 
sage system lOOb of the present invention is shown. 
The second embodiment of the multimedia message 
system 1 00b comprises an input/output communication 
device 112, the remote access system 1 08 and the mul- 
50 timedia message system 110. For convenience and 
ease of understanding like reference numbers are used 
for like components and have the same functionality as 
has been described above. In particular, the remote ac- 
cess system 108 and the multimedia message system 
55 110 are similar to those previously described. The sec- 
ond embodiment of the system 100b is different be- 
cause the input/output communication device 112 re- 
places the audio communication device 1 04 and the vis- 
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ual output device 1 06. The input/output communication 
device 112 is preferably a transceiver device that is ca* 
pable of: 1 ) receiving audio signals and converting them 
to sounds perceptible by a user 102; 2) receiving 
sounds, converting sounds to audio signal, and sending 5 
the audio signal, 3) generating control signals using an 
associated keypad or other input device; and 4) gener- 
ating an image viewable by a user 102. For example, 
the input/output communication device 112 is a cell 
phone having a LCD display and a message capability. io 
The input/output communication device 112 is coupled 
to the remote access system 1 08 by a bi-directional sig- 
nal line 130 to send and receive audio signals, send con- 
trol signals, and receive data and image displayable on 
the LCD display. For example, the signal line may be a ^5 
wireless communication link provided in a conventional 
manner for cell phones. While the present invention will 
be subsequently described primarily with reference to 
the first embodiment 100a, and not this embodiment 
100b, those skilled in the art will recognize how the 
present invention may also be adapted for used with the 
input'output communication device 112. 
[0036] Referring now to Figure 3 as well as Figures 1 
and 2, a block diagram of a preferred embodiment of the 
remote access system 108 for the message system -^5 
100a/l00b is shown. The remote access system 108 
preferably comprises a plurality of interfaces 302, 314, 
316, 318, a message retrieval unit 304, an object index- 
ing unit 306, a message composition unit 308. an image 
generation unit 310, a temporary cache or memory 311. 30 
and a database of users, user profiles, passwords, ob- 
jects, and list of attachable documents/objects 312. As 
has been noted above, the remote access system 1 08 
interconnects and provides the communication between 
the multimedia message system 1 1 0. the audio commu- 35 
nicatlon device 104 and the visual output device 106 or 
the input/output communication device 112. Essentially, 
the remote access system 1 08 translates the control in- 
puts from the audio communication device 104 into a 
format understandable by the multimedia message sys- 
tern 110, passes audio signals between the multimedia 
message system 1 1 0 and the audio communication de- 
vice 104, and passes visual signals to the visual output 
device 106. This is accomplished primarily by a data- 
base communication and retrieval unit 303 that is ^5 
formed of the message retrieval unit 304 and the mes- 
sage composition unit 308. The plurality of interfaces 
302, 314, 316, 318, the message retrieval unit 304, the 
object-indexing unit 306, the message composition unit 
308. the image generation unit 310. and the database so 
of users, user profiles, passwords, objects, and list of 
attachable documents/objects 312 are coupled to each 
other by a bus 320 in a conventional manner for passing 
data and control signals. The database 312 of users, 
user profiles, passwords, objects, and list of attachable 55 
documents/objects is preferably part of the multimedia 
message system 110, and is therefore, shown in Figure 
3 with dashed lines to represent an alternate embodi- 
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ment where the database 312 is part of the remote ac- 
cess system 108. 

[0037] The plurality of interfaces 302. 314. 316, 318 
are responsible for sending and receiving signals to and 
from respective other devices 104. 106, 110. 
[0038] The audio interface 302 is coupled to line 124 
and to the bus 320. The audio interface 302 preferably 
includes buffers for storing signals, as well as translators 
for converting audio signals on a telephone line to a dig- 
ital format usable by computers. The audio interface 302 
receives audio signal and control signals such as from 
a telephone handset, converts them to a digital format, 
and sends them to other units 304, 306, 308, 310 and/ 
or the multimedia message system 110 for further 
processing. The audio interface 302 could also include 
audio processing of a conventional type such as voice 
recognition and voice processing systems for voice 
mail. The audio interface 302 is also responsive to the 
other unit 304. 306, 308, 310 and the multimedia mes- 
sage system 110 to generate and send audio signals. 
[0039] The visual output interface 314 is coupled to 
line 126 and to the bus 320 for communication with the 
visual output device 1 06. The visual output interface 314 
preferably includes buffers for storing signals, as well as 
translators for converting data and control signals from 
a digital format to the format used by the visual output 
device 106. The visual output interface 314 also in- 
cludes hardware for communicating on the signal line 
126 conventionally used by the visual output device 1 06. 
Thus, the visual output interface 314 may include a net- 
work controller, a serial or parallel port controller, or a 
fax modem. The visual output interface 314 communi- 
cates and is primarily responsive to the image genera- 
tion unit 310. For example, the visual output interface 
314 converts signals and data from the image genera- 
tion unit 310 into any one of: 1) a fax signal if the visual 
output device 106 is a fax machine; 2) a printer control 
language file or postscript file if the visual output device 
1 06 is a directly connected printer; 3) a series of packets 
if the visual output device 1 06 connected via a network, 
or 4) properly formatted frames if the visual output de- 
vice 1 06 is directly connected CRT or flat panel display. 
[0040] In an alternate embodiment, the remote ac- 
cess system 108 may also include a visual input inter- 
face 318 for communication and coupling with a visual 
output device 106 that Includes the capability to gener- 
ate and send signals in addition to just receiving such 
signal. For example, a personal digital assistant (PDA) 
such as Palm brand computing devices or two-way pag- 
ers allows such bi-directional transfer of data and control 
signals. In such a case, the remote access system 1 08 
is adapted to include the visual input interface 318. The 
visual input interface 318 converts received signals and 
provides them to other units 304, 306, 308, 31 0 and the 
multimedia message system 110. Those skilled In the 
art will recognize that this is a simple conversion of the 
signals from a particular digital format to another. 
[0041 ] The multimedia message system interface 31 6 
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is coupled to line 1 28 and to the bus 320. The multimedia 
message system interface 31 6 preferably Includes buff- 
ers for storing signals, as well as translators for convert- 
ing the commands and data from the units 304. 306. 
308, 310 of the remote access system 108 to a proper 5 
input format for the multimedia message system 110. 
Inversely, the multimedia message system Interface 
316 Includes translators for converting the commands 
and data from multimedia message system 110 so that 
is usable by the units 304, 306, 308. 31 0 of the remote io 
access system 108. The multimedia message system 
interface 316 may communicate using a network proto- 
col or even as different streams or threads running on a 
single processor to pass information between the mul- 
timedia message system 110 and the units 304, 306, 
308, 310. 

[0042] In addition to the plurality of interfaces 302, 
314, 316, 318, the remote access system 108 Includes 
a number of units 304, 306. 308, 310 that enable remote 
access to the multimedia message system 110. The da- 
tabase communication and retrieval unit 303 is made up 
of the message retrieval unit 304 and the message com- 
position unit 308. These units 303, 304 and 308 control 
and handle the retrieval of messages and other infor- 
mation from the multimedia messaging system 110 and ^5 
delivery of such information to the audio interface 302 
and the visual output interface 314. These units 303, 
304 and 308 also control and handle the storage of mes- 
sages and other information in the multimedia messag- 
ing system 110 and the receipt of such messages and 30 
information from the audio interface 302 and the visual 
Input interface 318. In performing these operations, 
these units 303, 304 and 308 use the temporary cache 
311 for storing messages, objects, passwords, user 
identification numbers, etc. as needed before transfer 35 
to and from the interfaces. 302, 314, 316, 318 
[0043] The message retrieval unit 304 is also coupled 
to the bus 320 for communication with the audio inter- 
face 302, the multimedia message system interface 
316, the object-indexing unit 306. and the image gener- 40 
atlon unit 310. The message retrieval unit 304 interacts 
with these components as will be described below with 
reference to Figures 8, 9A, 9B, 10, 12A, 12B. and 13 to 
retrieve messages and their visual portions. The mes- 
sage retrieval unit 304 preferably controls the interaction 45 
and signal passing between the audio interface 302 and 
multimedia message system interface 316 for passing 
the audio signals. The message retrieval unit 304 Is also 
responsive to control signals received via the audio in- 
terface 302 to undertake additional steps and solicit 50 
more information from the user, necessary to identify 
and deliver audio message portions or objects as well 
as visual message portions or objects. The message re- 
trieval unit 304 also processes signals from the multi- 
media message system interface 316 and provides the 55 
information necessary for the image generation unit 310 
to work with the muttimedia message system 1 1 0 and 
the visual output interface 314 and send images to the 



user 102. 

[0044] The object-Indexing unit 306 Is also coupled to 
the bus 320 for communication with the multimedia mes- 
sage system interface 316. the message retrieval unit 
304 and the message composition unit 308. The object- 
indexing unit 306 interacts with these components as 
will be described below with reference to Figures 8, 9A, 
9B, 10, 12A, 128, and 13 to translate signals input via 
the audio interface 302 to corresponding commands 
and references to objects that are portions of each mes- 
sage. The object-indexing unit 306 is also used to gen- 
erate and store unique indices that become part an im- 
age of the message. The object-indexing unit 306 pref- 
erably stores these indices for current and future use in 
accessing the messages portions such as in trie data- 
base 312, or in a database of the multimedia messaging 
system 110. This is described in more detail below with 
reference to Figures 11A and 11B. These Indices are in 
turn input by the user 102 to specify commands and 
identify objects corresponding to the commands. Other 
units 304, 308, 310 and interfaces 302, 314, 316 use 
the object-indexing unit 306 to translate indices to spe- 
cific actions and objects or files. 
[0045] The message composition unit 308 is also cou- 
pled to the bus 320 for communication with the audio 
interface 302, the multimedia message system interface 
316, and the object-Indexing unit 306. The message 
composition unit 308 interacts with these components 
as will be described below with reference to Figure 14 
to create new messages. Using audio input and con- 
trols, the user 1 02 can create reply or original messages 
Including the inclusion of pre-existing objects (video 
clips, documents. Images, etc.). The message compo- 
sition unit 308 translates the data and control signal re- 
ceived via the audio interface 302 using the object-in- 
dexing unit 306 into commands and data understanda- 
ble to the multimedia message system 110. The multi- 
media message system 110 in response is able to cre- 
ate and store such remotely created messages. 
[0046] The image generation unit 310 is responsive 
to the message retrieval unit 304 and receives and 
sends data and images from the multimedia message 
system 110 to the visual output interface 314. The op- 
eration of the image generation unit 310 can be best un- 
derstood with reference to Figures 9B. 10 and 13. The 
image generation unit 310 communicates with the mul- 
timedia message system 1 1 0 to receive visual message 
objects or portions and Is responsible for the delivery of 
the objects to the user 102 according to the type of visual 
output device that the user 102 has specified or has 
available. 

[0047] The database of users, user profiles, pass- 
words, objects, messages and list of attachable docu- 
ment/objects 312 is shown as being connected to the 
bus 320 for access by the various units 304, 306, 308, 
310 and interfaces 302, 314. 316. Those skilled in the 
art will recognize that this information is preferably 
stored as part of the multimedia message system 110, 
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and thus be accessible through the multimedia mes- 
sage system interface 316. Those skilled the an will re- 
alize that this database 31 2 may be part of the multime- 
dia message system 110 and cached locally as part of 
the remote access system 1 08. The database 312 pref- 5 
erably stores user identification numbers and pass- 
words used for comparison and authorization for remote 
access, objects or those constituent portions that form 
the messages, and list of attachable objects or docu- 
ments. Such a list of attachable objects or documents 
also has object indices for remotely using the objects. 
[0048] Referring now to Figures 4A-4E, various ex- 
emplary embodiments of possible configurations of the 
remote access system 108 and various conventional 
devices as the audio communication device 1 04 and the ^5 
visual output device 106 are shown. These figures are 
used to illustrate various different ways in which the 
methods of the present invention may be used In whole 
or pan to access any part of a multimedia message. 
[0049] Figure 4A illustrates an example where the au- 
die communication device 104 is a conventional tele- 
phone and there is no visual output device 106. In such 
a case, the user 102 may have printed an image of a 
message on paper 402 using a computer or other ap- 
paratus that is part of the multimedia message system 25 
110. The user may then leave the office and take the 
paper 402. Using only the telephone 104 and the paper 
402. the user 102 may access the remote access sys- 
tem 1 08 to retrieve the message. Since there is no visual 
output device 106. user 102 is only able to listen to or 30 
add audio portions of the message. 
[0050] Figure 4B Illustrates an example where the au- 
dio communication device 104 is a telephone and the 
visual output device 106 is a printer 404. The printer 404 
is linked in some conventional manner to the remote ac- 35 
cess system 108. In this situation, the user 102 is as- 
sumed to be away from the office, and thus does not 
have normal access to the multimedia message system 
1 1 0. but is at a remote location that has a telephone 1 04 
and a printer 404, The user 102 is able to access the 40 
multimedia message system 110 for audio messages 
via the telephone 104. The user 102 is also able to re- 
trieve copies of visual portions of the message by print- 
ing them on the printer 404. For example, the paper 402 
having an image of the message may be printed and ^5 
used to access additional portions of the message as 
will be described in more detail below. 
[0051 J Figure 4C illustrates an example where the au- 
dio communication device 104 is a telephone and the 
visual output device 106 is a fax machine 406. A tele- so 
phone line couples the fax machine 406 in a conven- 
tional manner to the remote access system 108. In this 
situation, the user 102 is assumed to be away from the 
office such as at a hotel. The user 1 02 is able to remotely 
access the functionality provided by the multimedia 55 
message system 110. The user 102 is able to access 
the multimedia message system 110 for the creation, 
retrieval and deletion of audio messages via the tele- 



phone 104. The user 102 is also able to retrieve copies 
of visual portions of the message by having then faxed 
to the fax machine 406. Both the image of the message, 
including indices, as well as the non-audio portions or 
objects contained in the message can be faxed by the 
multimedia message system 11 0 to the fax machine 406 
for view by the user 1 02. The fax machine 406 may also 
be used to send visual objects such as images and doc- 
uments to the multimedia messaging system 110. Once 
input to the multimedia messaging system 110, these 
objects may be resent, added, deleted or similarly ma- 
nipulated just like other visual objects already stored in 
the system. 

[0052] Figure 4D illustrates an example where the au- 
dio communication device 104 is a telephone and the 
visual output device 106 Is a hand-held computing de- 
vice 408 that includes wireless communication capabil- 
ity. For example, the hand-held computing device 408 
could be a personal digital assistant (PDA) such as the 
Palm VM by 3Com. The hand-held computing device 
408 communicates with the remote access system 108 
via an RF link 126. In this case, the user 102 is assumed 
to be away from the office but has access to a telephone 
and carries their hand-held computing device 408 with 
them. The user 102 is able to remotely access the func- 
tionality provided by the multimedia message system 
110 using the telephone to retrieve audio message por- 
tions, send audio messages and send control signals. 
With the display portion of the hand-held computing de- 
vice 408. the user 102 is able to view and to retrieve 
copies of visual portions of the message. Furthermore, 
the user 1 02 may use the input mechanism of the hand- 
held computing device 408 to send control signals and 
visual objects to the remote access system 108. When 
using a PDA 408 or other interactive display with a tel- 
ephone 104 for creating new messages, the user 102 
can indicate a reference by tapping the reference on the 
display while recording audio on the phone. The system 
1 00a advantageously keeps track of the timing and uses 
it for creating references. Examination of the audio con- 
tent might be combined with the timing (e.g. "this docu- 
ment") to Improve reference location. Furthermore, the 
gestures or other inputs provided by a PDA 408 could 
be translated into any variety of control signals as will 
be understood by one skilled in the an. It should also be 
noted that while the methods are described below pri- 
marily with reference to use with passive devices such 
as printers and fax machines, the user interface and pro- 
cedures of the methods are slightly different for interac- 
tive displays (which might initiate document retrieval 
events, and other control inputs) as will be understood 
to those skilled in the art. 

[0053] Figure 4E illustrates an example where the au- 
dio communication device 104 is a telephone and the 
visual output device 106 is a computer 410 connected 
to the remote access system 1 08 by the Internet or direct 
dial in. In this case, the user has left the office, but has 
a telephone and possibly a home computer 410 to ac- 
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cess the multimedia message system 11 0. The tele- 
phone 104 is used for the audio portions of messages, 
and the display of the computer 410 can be used to ob- 
tain the visual message portions. If the computer display 
is a web browser, clicking on "hyperlinks" within a dis- 
play of the computer 410 could have the same control 
and reference function (e.g., "this document") as tap- 
ping the PDA 408 in Figure 4D. 
[0054] Figure 5 is a block diagram of a third and pre- 
ferred embodiment of the multimedia message system 
120 constructed in accordance with the present inven- 
tion. In this preferred embodiment, the remote access 
system is integrated as a part of the multimedia mes- 
sage system. More particularly, the remote access ca- 
pability is provided by adding software for operation of 
the multimedia messaging with the functionality that has 
been described above. 

[0055] Some portions of the detailed descriptions that 
follow are presented in terms of algorithms and symbolic 
representations of operations on data bits within a com- 
puter memory 608. These algorithmic descriptions and 
representations are the means used by those skilled in 
the data processing arts to most effectively convey the 
substance of their work to others skilled in the art. An 
algorithm is here, and generally, conceived to be a self- 
consistent sequence of steps leading to a desired result. 
The steps are those requiring physical manipulations of 
physical quantities. Usually, though not necessarily, 
these quantities take the form of electrical or magnetic 
signals capable of being stored, transferred, combined, 
compared, and otherwise manipulated. It has proven 
convenient at times, principally for reasons of common 
usage, to refer to these signals as bits, values, ele- 
ments, symbols, characters, terms, numbers, or the like. 
[0056] It should be borne in mind, however, that all of 
these and similar terms are to be associated with the 
appropriate physical quantities and are merely conven- 
ient labels applied to these quantities. Unless specifical- 
ly stated othenwise as apparent from the following dis- 
cussion, it is appreciated that throughout the descrip- 
tion, discussions utilizing terms such as "processing" or 
"computing" or "calculating" or "determining" or "dis- 
playing" or the like, refer to the action and processes of 
a computer system, or similar electronic computing de- 
vice, that manipulates and transforms data represented 
as physical (electronic) quantities within the computer 
system's registers and memories into other data simi- 
larly represented as physical quantities within the com- 
puter system memories or registers or other such infor- 
mation storage, transmission or display devices. 
[0057] The present invention also relates to an appa- 
ratus for performing the operations herein. The appara- 
tus may be specially constructed for the required pur- 
poses, or it may comprise a general-purpose computer 
(See Figures 5. 6A, 6B) selectively activated or recon- 
figured by a computer program stored in the computer. 
Such a computer program may be stored in a computer 
readable storage medium, sLK:h as. but not limited to, 
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any type of disk including floppy disks, optical disks. CD- 
ROMs, and magnetic-optical disks, read-only memories 
(ROMs), random access memories (RAMs). EPROMs, 
EEPROMs. magnetic or optical cards, or any type of me- 

5 dia suitable for storing electronic instructions, and each 
coupled to a computer system bus. 
[0058] The algorithms and displays presented herein 
are not inherently related to any particular computer or 
other apparatus. Various general-purpose systems may 

TO be used with programs in accordance with the teachings 
herein, or it may prove convenient to construct more 
specialized apparatus to perform the required method 
steps. The required stmcture for a variety of these sys- 
tems will appear from the description below. In addition, 

f5 the present invention is not described with reference to 
any particular programming language. It wilt be appre- 
ciated that a variety of programming languages may be 
used to implement the teachings of the invention as de- 
scribed herein. 

20 [0059] Refenring also to Figure 6A is a block diagram 
of a preferred embodiment of a multimedia message 
system 120 including remote access capability is shown 
in more detail. The remote capable multimedia message 
system 120 preferably comprises an audio interface 

25 602, a processing unit 604, a data storage device 606, 
a memory unit 608, a display device 61 0, a network con- 
troller 612, an input device 614, a visual output interface 
616. and optionally a visual input interface 618 all of 
which are communicatively coupled to system bus 620. 

30 [0060] The processing unit 604 processes data sig- 
nals and may comprise various computing architectures 
including a complex instruction set computer (CISC) ar- 
chitecture, a reduced instruction set computer (RISC) 
architecture, or an architecture implementing a combi- 

35 nation of instruction sets. Although only a single proc- 
essor is shown in Figure 6A, multiple processors may 
be included. 

[0061] Data storage device 606 stores data and in- 
structions for processing unit 604 and may comprise 
40 one or more devices including a hard disk drive, a floppy 
disk drive, a CD-ROM device, a DVD-ROM device, a 
DVD-RAM device, a DVD-RW device, a flash memory 
device, or some other mass storage device known in the 
art. 

45 [0062] The display device 610 comprises any device 
equipped to display electronic images and data as de- 
scribed herein. Display device 610 may be, for example, 
a cathode ray tube (CRT), liquid crystal display (LCD), 
or any other similarly equipped display device, screen, 

50 or monitor. In one embodiment, display device 610 is 
equipped with a touch screen in which a touch-sensitive, 
transparent panel covers the screen of display device 
610. 

[0063] The input device 614 is preferably a keyboard 
55 coupled to communicate information and command se- 
lections to the processing unit 604. and a cursor control 
device equipped to communicate positional data as well 
as command selections to the processing unit 604. The 
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cursor control device may include a mouse, a trackball, 
a stylus, a pen, cursor direction keys, or other mecha- 
nisms to cause movement of a cursor. 
[0064] The network controller 612 links processing 
unit 604 to a network that may include multiple process- 
ing systems. The network of processing systems may 
comprise a local area netv/ork (LAN), a wide area net- 
work (WAN) (e.g., the Internet), and/or any other inter- 
connected data path across which multiple devices may 
communicate. 

[0065] System bus 620 represents a shared bus for 
communicating information and data-throughout the 
multimedia message system 120. System bus 620 may 
represent one or more buses including an industry 
standard architecture (ISA) bus. a peripheral compo- 
nent interconnect (PCI) bus, a universal serial bus 
(USB), or some other bus known in the art to provide 
similar functionality. 

[0066] The audio interface 602. visual output interface 
616 and optionally visual input interface 61 8 have been 
described above with reference to Figure 3. This de- 
scription will not be repeated here. These interfaces 
602, 616, 618 provide the same functionality for com- 
munication and interaction with the processing unit 604. 
[0067] Memory unit 608 may store instructions and/or 
data that may be executed by processing unit 608. The 
instructions and/or data may comprise code for perform- 
ing any and/or all of the techniques described herein. 
Memiory unit 608 may be a dynam.ic random access 
memory (DRAM) device, a static random access mem- 
ory (SRAM) device, or some other memory device 
known in the art. 

[0068] Figure 6B shows the memory unit 608 of the 

multimedia message system 120 of Figure 6A in more 
detail. The memory unit 608 preferably comprises an 
operating system 652, a multimedia message manage- 
ment module 654. a message creation module 656, a 
message editing and deletion module 658, a memory 
660 storing user identification numbers, passwords, ob- 
jects identification numbers, and list of attachable ob- 
jects, an object indexing module 662. a remote message 
retrieval module 564, a remote image generation mod- 
ule 656 and a remote composition module 668. These 
modules 652-668 are coupled by bus 620 to the 
processing unit 604 for communication and cooperation 
to provide the system 120 for multimedia messaging 
that further provides a capability for the user to access 
messages using a conventional telephone or any avail- 
able visual output device 106. 

[0069] The operating system 652 Is preferably one of 
a conventional type such as, WINDOWS®, SOLARIS® 
or LINUX® based operating systems. Although not 
shown, the memory unit 608 may also include one or 
more application programs including, without limitation, 
word processing applications, electronic mail applica- 
tions, spreadsheet applications, and web browser appli- 
cations. 

[0070] The multimedia message management mod- 



ule 654. the message creation unit 656. the message 
editing and deletion module 658. in this embodiment, 
are programs or instructions for controlling the process- 
ing unit 604 to provide the basic functionality of a multi- 

5 media message system. The functionality provided by 
these modules is described in pending U.S. Patent Ap- 
plication Serial No. 09/407,010, entitled "Method and 
Apparatus for Generating Visual Representations for 
Audio Documents," filed on September 28. 1 999, as has 

10 been noted above. The multimedia message manage- 
ment module 654 is. different in that it also controls the 
interaction of these modules 656. 658 with the remote 
access modules 662, 664, 666, 668. This interaction 
and control will be described below along with the flow- 

15 charts. The remaining modules 656 and 658 are also 
responsive the remote access modules 662, 664, 666, 
668 as necessary to create, edit delete and retrieve mul- 
timedia messages. 

[0071] The object-indexing module 662, the remote 
20 message retrieval module 664, the remote image gen- 
eration module 666 and the remote message composi- 
tion module 668 are similar in functionality to the corre- 
sponding units 306, 304, 310 and 308 that have been 
described above. However, the modules 662, 664, 666, 
25 668 are programs or instructions executable by the 
processing unit 604 to provide the remote functionality 
as noted above and as will be fully detailed with the flow- 
charts of Figures 8-1 4B. 

[0072] Referring now to Figure 7, a graphical repre- 

30 sentation of a multimedia message or chronicle 700 is 
shown. The ability to generate such an image of the mul- 
timedia message 700 is a key aspect of the present in- 
vention because it enables remote access. The viewa- 
ble or printable image of the message 700 preferably 

35 comprises a number of informational components such 
as a message header 702, a remote message access 
information block 704. a plurality of message portions 
or objects graphically depicted 706a, 706b, 708, 710, 
712, 714, a key of action-to-index correspondence 716, 

40 one or more links 720 between message objects 706a, 
706b, 708, 710, 712, 714, and one or more indices 730 
corresponding to each message portion 706a, 706b, 
708, 710. 712. 714 that is remotely retrievable. By pro- 
viding a graphical view of the image, the present inven- 

45 tion is particularly advantageous because the user 1 02 
is able to more readily understand the context and in- 
terrelationship between the messages portions or ob- 
jects. 

[0073] The message header 702 preferably provides 
50 basic information about the message 700 such as who 
the message is from, the subject of the message and 
the date when the message was last saved. This infor- 
mation is similar to that you would find for e-mail mes- 
sages and provides a convenient basis for the user 102 
55 to identify a message in which they are interested. 
[0074] The remote message access information block 
704 provides information essential for the user 102 to 
access the message remotely. In particular, the remote 
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message access Information block 704 includes the tel- 
ephone number that the user must dial to gain access 
to the multimedia message system 110. The remote 
message access information block 704 also includes a 
unique user identification number and a unique mes- 
sage identification number that the user 102 can enter 
via the keypad of the telephone to indicate to the system 
1 00a who the user is. and the specific message that the 
user 102 wants to access. In alternate embodiments, 
the remote message access information block 704 may 
contain a URL or a barcode that could be faxed or other 
index that could be input via the telephone, fax machine 
or PDA to indicate the message that the user 1 02 wants 
to access. The remote message access information 
block 704 may also contain other information for navi- 
gating or accessing the system 100a such as how to 
reply to a message as shown in Figure 7. 
[0075] The plurality of message portions or objects 
706a, 706b, 708. 710, 712, 714 are preferably graphi- 
cally depicted, and thumbnail images are created for 
each type. Since the message 700 may include any one 
or groups of audio messages, documents, photographs, 
images, and video clips, the system lOOa preferably a 
generates a thumbnail image of each so that the user 
102 can immediately identify the type of objects that 
form the message. The example shown in Figure 7 has 
a message portion of each type including two audio por- 
tions. The audio portions 706a. 706b are preferably 
graphically depicted as part of the image in a manner 
similar to how they would be displayed on a computer 
display that was not remotely connected. The audio por- 
tions 706a, 706b preferably include an image of the per- 
son leaving the message, an icon representing a relative 
duration of the audio message. Document 708 Is repre- 
sented by a thumbnail image of the first page of the doc- 
ument. Such documents 708 may simply be text or may 
be displayed in proprietary application formats, as the 
user would see the first page of the document. Photo- 
graphs 710 and images 714 are represented by a 
thumbnail image of the photograph. Video clips are rep- 
resented by a thumbnail image of a frame of the video 
clip. Thus, as can be seen in Figure 7, simply by viewing 
the multimedia message 700, the user 102 is able to 
identify what type of message portions are contained in 
the message. 

[0076] The multimedia message 700 preferably in- 
cludes a key of action-to-index correspondence 716. 
The key 716 identifies to the user 102 what actions will 
be taken by the system 1 00a in response to the Input of 
an index. Such actions include but are not limited to ^ 
playing an audio stream, faxing a displayed page of a 
documents, faxing an entire document, faxing an image 
at normal size and faxing an image zoomed to fill a sin- 
gle page. 

[0077] The multimedia message 700 also advanta- 5 
geously displays links 720 between message objects 
706a. 706b. 708. 710, 712 and 714. These links 720 
convey to the user 102 the relationship of documents 



708. images 710. 714 and video clips 712 to the audio 
portions of the message 706a. 706b. 
[0078] The most important part of the multimedia 
message 700 for enabling remote access Is the Indices 
5 730 corresponding to each message portion 706a, 
706b, 708, 710. 712 and 714. Each message portion 
706a, 706b, 708, 710, 712 and 714 that is remotely re- 
trievable is provided a set of indices 730 that are adapt- 
ed for the types of delivery suited to the type of message 
10 portion 706a. 706b, 708, 71 0. 712 and 714. The present 
invention advantageously generates unique indices 730 
and positions them near the upper left comer of the 
thumbnail image of the message portion 706a, 706b, 
708. 710, 712 and 714. The unique indices 730 prefer- 
15 ably have two components. A first component 732 indi- 
cates the type of action that will be taken by the system 
100a if the user 102 inputs the index as part of the proc- 
ess of retrieving a message ponion 706a, 706b. 708, 
710, 712 and 714. The second component 734 Is a 
^0 number that uniquely identifies which message portion 
upon which the action will be taken . This is just one em- 
bodiment for generating the indices and those skilled in 
the art will recognize that a variety of other indices may 
be used. For the exemplary message 700 of Rgure 7, 
there are five possible actions, and six message por- 
tions. As can be seen for the audio messages portions 
706a, 706b, there is only one way to output them and 
each is uniquely identified, thus each only has one in- 
dex. However, for the Images, document and messag- 
30 es. each has two delivery options, thus each portion 
708, 710. 712 and 714 is provided with two indices 
based on whether It Is an Image or a document. 

Overview 

35 

[0079] While the methods of the present invention will 
now be described with reference to methods in which 
the user is prompted to provide input, it should be noted 
that the preferred methods of the present Invention are 

^o preferably event-based methods, where the user is able 
to input control signals at any time and the system will 
execute processes and methods In response to input of 
such control signals. Therefore, in various alternate em- 
bodiments of the present Invention different levels of 

'5 prompting may be used, and those skilled in the art will 
understand how the methods detailed with reference to 
Figures 8-1 6 could be adapted to provide such differerit 
levels of linear prompting. 

[0080] Refenrlng now to Figure 8. there is shown a 
'0 flow diagram of a top-level control process for interact- 
ing with the multimedia message system lOOa of the 
present invention. This flowchart shows a typical way 
that a user 102 would Interact from a remote location 
with the multimedia message system lOOa. Processing 
5 begins in step 802 with the user 1 02 accessing the mul- 
timedia message system lOOa and retrieving a mes- 
sage or the audio portions of it. Then the system lOOa 
generates in step 804 an image of the message 700 that 



10 

X^ID: <EP 10g4656A2J_> 




EP 1 094 



includes the indices 730 as described in Figure 7. This 
process is more fully detailed below in Figures 9A and 
9B. Once the user 102 has the image of the message 
700. the user 1 02 is able to retrieve 806 any portions of 
the message 700 desired using the indices 730. Once 5 
the user 102 has received the portions of the message 
700 desired, the user 102 can remotely create 808 a 
response message that may include an audio portion 
and other portions, and send it to another user or add it 
to a message. 

Retrieving An Image Of A Multimedia Message 

[0081] Figures 9A and 9B show a flow diagram of a 
preferred method for retrieving an image of a multimedia 
message 700 in accordance v/ith the present invention. 
The process begins in step 902 when the system 100a 
receives a telephone call from the user 102. The user 
102 initiates the process by dialing a predefined tele- 
phone number provided to the user 1 02 by a system ad- 20 
ministrator similar to off-site use of voice mail systems. 
The system lOOa then requests 904 a user ID number 
by generating audio signals that are voice prompts and 
sending them to the user 102 over the telephone line. 
In an alternate embodiment, this could be done auto- ^5 
matically using caller ID or automatic number identifica- 
tion, and the need for requesting a user ID could be elim- 
inated. The methods of the present invention will now 
be described below as interacting with the user 102 and 
the user 102 providing input. Unless otherwise speci- 30 
fied, the user 102 interfaces with the system 100a in a 
manner conventional to existing voice mail systems. 
The user 102 inputs his ID number using the keypad of 
the telephone. The system 1 00a receives 906 the user's 
ID number, and determines 908 if the user's ID number 35 
is valid. This done by a comparison of the number input 
to a database of valid numbers. If the number is not val- 
id, the method signals 910 an error and ends. If the 
number is valid, the method continues to prompt 91 2 for 
a password. The user 1 02 inputs his password using the 
keypad of the telephone, and the password is received 
914 by the system 100a. The system 100a then tests 
91 6 if the password is valid. If the password is not valid, 
an error is signaled 918 and the process ends. Other- 
wise, the process continues to determine 920 if the user ^5 
has ended the call by hanging up. At any time, the user 
102 can terminate the process by ending the call. If the 
user 102 has ended the call, the process ends. 
[0082] If the user 1 02 has not ended the call, the proc- 
ess continues to step 922. In step 922, the system 1 00a so 
prompts the user 1 02 to listen to the next new message 
by entering the # sign on the telephone 104 or to listen 
to a specific message by inputting the message ID 
number using the keypad ofthe telephone 104. The user 
102 Inputs signals using the keypad of the telephone 55 
102, and the signals are received 924 by the system 
100a. Then . in step 925, the system I00a retrieves ei- 
ther the next new message if the user input the # sign 
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or the specific message indexed if the user a message 
ID number. Then the system 1 00a plays a first audio por- 
tion of the message for the user 1 02. The system 1 00a 
preferably designates one audio portion in each mes- 
sage as a default to be played to the user 102 In step 
926. Once the message has be played for the user 1 02, 
the system 1 00a prompts 928 the user 1 02 as to wheth- 
er he would like a visual version of the message to be 
sent by the system 1 00a. For example, the user 1 02 can 
be prompted to enter a particular numerical combination 
If a visual version of the message Is to be sent by the 
system 1 00a. Alternatively, a predetermined input such 
as the key could be input at any time the message 
was being played 926 as a default input for requesting 
a visual copy of the message. In the alternate case, the 
message would be sent to default location by a default 
means that are pre-stored in a user profile. Then the 
system 100a determines 930 whether the user 102 
wants a visual version of the message to be delivered. 
If the user 1 02 does not want a visual version of the mes- 
sage to be delivered, the method returns to step 920 to 
determine if the call has been ended. 
[0083] If the system 1 00a determines in step 930 that 
the user 1 02 wants a visual version of the message, the 
method proceeds to step 934. In step 934, the system 
100a determines the mechanism for delivering the vis- 
ual version of the message, and receives information 
from the user for delivery ofthe message. How the mes- 
sage is delivered is dependent on the type of visual out- 
put device 106 that the user 102 has access to. This 
step of determining the delivery mechanism is described 
in detail below with reference to Figure 10. After the de- 
livery means has been determined, the system 100a 
generates 936 an annotated image of the message. The 
annotated image preferably includes indices of the ob- 
jects contained in the message 700 as has been de- 
scribed above with reference to Figure 7. This process 
is more fully detailed below with reference to Figures 
11 A and 11B. Once the annotated message has been 
generated 936, the process completes by sending 938 
the annotated message to the user 102. Depending on 
the delivery mechanism, this may be by faxing; sending 
an image, or printing. After the visual Image of the mes- 
sage has been sent, the process returns to step 920 to 
determine if the call has been ended. 
[0084] Those skilled in the art will recognize that there 
are a number of different ways to retrieve visual repre- 
sentations of new messages in addition to that just de- 
scribed with reference to Figures 9A and 9B. One way 
is for a user 1 02 to call the system 1 00a and specify a 
fax number. (Alternatively, the user need only call into 
the system 100a: and the system 100a can Identify the 
user using caller ID or automatic number identification, 
retrieve the user's profile from the database 312 for a 
default location and for the format for delivering visual 
objects.) The system 1 00a in response identifies the us- 
er's messages, and generates an overview cover sheet 
that has a thumbnail representation for each message 
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(including visual representations for plain "voice" mes- 
sages). This overview cover sheet functions just like all 
other pages the thumbnails have indices allowing 
them to be retrieved or referenced via phone. Also, de- 
pending on the user's preferences, the overview cover 
sheet might be followed by one page for each full mul- 
timedia message (e.g., not just a voice message). More- 
over, the user 102 can reply to the messages on the 
overview cover sheet by entering a reply command and 
the index for the message. In yet another embodiment, 
the generation of the overview cover sheet can be done 
automatically in response to the user 1 02 calling into the 
system 100a. the userl02 receiving a message, or any 
other event. In that case, the overview cover sheet is 
generated in response to the triggering event and sent 
to the user 1 02 via default delivery mechanism to a de- 
fault location. 

Determining The Delivery Mechanism 

[0085] Figure 1 0 is a flow diagram of a preferred meth- 
od for determining the delivery mechanism for the an- 
notated image of the message. The method for deter- 
mining the delivery mechanism begins by determining 
1 002 whether the visual output device 106 is a facsimile 
machine. This is preferably accomplished by prompting 
the user 102 to indicate if the visual output device 106 
is a facsimile machine. If the user 1 02 responds that the 
visual output device 106 is a facsimile machine, the 
method continues and prompts 1004 the user 102 to in- 
put the telephone number for the facsimile machine. 
Then the method receives the input from the user 102. 
sets 1006 the delivery mechanism to be via facsimile 
transmission to the number received from the user 102. 
The method also stores the facsimile number for future 
use, after which the method ends. 
[0086] If the delivery mechanism is determined not to 
be by facsimile, then the method tests 1008 whether de- 
livery mechanism is by sending e-mail. This is again ac- 
complished by prompting the user 102 to indicate if the 
visual output device 106 is a computer. If the user 102 
responds that the visual output device 106 is a compu- 
ter, the method continues and prompts 1010 the user 
102 to input the e-mail address where the messages or 
message portion should be sent. Then the method re- 
ceives the input from the user 102, sets 1012 the deliv- 
ery mechanism to be by e-mail to the address received 
from the user 102. Alternatively, the method could auto- 
matically retrieve an e-mail address from a profile for the 
user, thus, obviating the need for the user to repeatedly 
provide his/her e-mail address. The method may also 
store the email address for future use. after which the 
method ends. 

[0087] If the delivery mechanism is determined not to 

be by e-mail, then the method tests 1014 whether de- 
livery mechanism is by printing to an identified printer. 
If the delivery mechanism is by printing, the method con- 
tinues to prompt 1016 the user 102 for a printer name 
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or some other input to identify the address of the printer. 
In one embodiment, the system could prompt the user 
1 02 with various printer names and request that the user 
input particular keys on the keypad of the audio com- 

5 munication device 1 04 when the name of the desired 
printer is given. Once the user 102 has responded with 
input, the method receives the input, sets 1018 the de- 
livery mechanism to be by postscript file to a specific 
printer, and store the printer name for future use. 

10 [0088] If the delivery mechanism is detennined not to 
be by printer, then the method tests 1020 whether de- 
livery mechanism is sending to a display device. If the 
user confirms that the delivery mechanism is sending to 
a display device, the method proceeds to set 1022 the 

15 delivery for output to a conventional display device such 
as a CRT or LCD panel, after which the method is com- 
plete. If the delivery mechanism is not sending to a dis- 
play device, the system 1 00a signals 1 024 and error be- 
cause the user 102 has failed indicate the method for 
^0 sending the visual images to the user. 

Generating An Image of a Message 

[0089] Referring now to Figures 11 A and 11 B, a pre- 

^5 ferred method for generating a remotely usable image 
of a multimedia message Is shown. The method begins 
with the identification of a multimedia message by the 
user Then the system 100a determines 1102 the ob- 
jects or message portions that are contained in the mul- 

30 timedia message. A first one of the objects determined 
to be part of the message is then retrieved, and an image 
of the retrieved object is generated 1106. As noted 
above, the image will depend on the type of object, and 
is preferably a thumbnail image of some portion of the 

35 object's content. Audio portions will have one format, im- 
ages and documents will have a second format, and vid- 
eo clips with have yet a third different format. Once the 
image is created, a reference or object number will be 
assigned 1108 to the image and the object. The refer- 

40 ence numbers are preferably assigned and stored for 
later retrieval as each image is created such as using a 
counter to keep track. 

[0090] The method next determines 1110 whether the 
retrieved object is an audio clip. If the retrieved object 

45 is an audio clip, the method generates 1 1 1 2 an index for 
the object based on the object number (Step 1108) and 
a number associate with the action of playing an audio 
message. For example, the present invention provides 
the number "1" as indicating an action of playing a mes- 

50 sage. Thus, for an audio message a single index is cre- 
ated that is a combination of 1 and the object number. 
After generation of the indices, the method continues in 
step 1 1 28. While the present invention contemplates on- 
ly a single action for audio messages, altemate embod- 

55 iments could specify any number of other actions be per- 
formed with the audio portion or object. 
[0091 ] If the retrieved object is not an audio clip 1110, 
the method next tests 1 1 1 4 whether the retrieved object 
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is a document. If the retrieved object Is determined to 
be a document, the method generates 1116 indices for 
the object using object number generated in step 11 08 
and the output options for documents. In the preferred 
embodiment, the method has two output options for doc- 5 
uments: 1 ) to send the entire document to the visuat out- 
put device 106. or 2) to sent only the first page of the 
document to the visual output device 1 06. Thus, for each 
document object, a pair of indices will be created. In an 
exemplary embodiment, the number "2" is input if the io 
user 1 02 wants only the first pages sent, and the number 
"3" is input if the user 102 wants the entire document 
sent. Therefore, the first index for a document is 2 and 
the object number concatenated, and the second index 
for the same document Is 3 and the object number con- ^5 
catenated. The indices are generated 1116 forthe object 
after which the method continues in step 1128. Those 
skilled in the art with recognize that there could be a 
varieiy of other output options for documents in alternate 
embodiments such as sending the first and last page of 
the document. 

[0092] If the retrieved object is not a document 1114, 
the method next determines 1118 whether the retrieved 
object is an image. If the retrieved object is determined 
to be an image, the method generates indices for the ^5 
object using object number generated in step 1108 and 
then outputs the options for images. In the preferred em- 
bodiment, the method provides two output options for 
images: 1 ) to send the image at normal size to the visual 
output device 1 06. or 2) to zoom the image to fit an out- 30 
put page of the visual output device 106 and send the 
zoomed image to the visual output device 1 06. Thus, for 
each image object, a pair of Indices will be created. In 
an exemplary embodiment, the number"4" is input if the 
user 1 02 wants only the image at normal size sent, and 35 
the number "5" Is input if the user 1 02 wants a zoomed 
image sent. Therefore, the two Indices for images are 4 
in combination with the object number, and 5 in combi- 
nation with the object number. The indices are generat- 
ed 1120 forthe object after which the method continues "fo 
in step 1128. Again, those skilled in the art will under- 
stand that there could be a variety of other output op- 
tions for images in alternate embodiments, 
[0093] If the retrieved object is not an image 1 11 8, the 
method next determines 1 1 22 whether the retrieved ob- ^5 
ject is a video clip. If the retrieved object is determined 
to be a video clip, the method generates 1124 indices 
for the object using object number generated in step 
1108 and the output options for video clips. This step 
could be the same as step 1120 if only the two output 50 
options are provided. However, if a different output op- 
tion such as displaying selected frames of the video clip 
Is provided, then steps 1 1 22 and 1 1 24 generate the cor- 
responding indices. After step 1124, the method contin- 
ues to step 1128. 

[0094] If the retrieved object is not a video clip 1122, 
the method proceeds to step 1 1 26. In one embodiment, 
an error is Indicated to the user 1 02 to Indicate that only 
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some of the message portions will be retrievable before 
proceeding to step 1128. Alternatively, there may be 
some portions or objects in a message that are not pro- 
vided with an index, are not retrievable, and thus, only 
are shown as thumbnails in the message 700. While not 
shown in Figure 11 A, the present invention could be 
modified to test for and create Indices for . any other 
types of objects. For example, if web pages are provided 
as a type of object, this method could include the steps 
of testing for whether an object Is a web page and gen- 
erating Indices based on the method for outputting web 
pages. Those skilled in the art will recognize how this 
may be extended for any other object types. 
[0095] Those skilled in the art will recognize that steps 
1112, 1116, 1120, and 1124 provide only one scheme 
for generating indices based on primarily on each index 
having portions specifying the function and a document 
number. There are a variety of other indices generation 
methods that could also be used in the present Inven- 
tion. For example, the indices generation could be 
based primarily on user ID and document type. Such a 
scheme would have portions of the index formed by user 
ID, object type and a number of Images from the author 
(e.g. #20301 might indicate author No. 2, image data 
type, and 1 Image added by this author). Such a trans- 
parent numbering or indexing scheme allows users to 
remember and enter indices. This even allows access 
to the message without need for a visual Image of the 
message. For example, a user could just input Indices 
based on know user ID and type of messages desired. 
The system 100a would either return the messages or 
indicate that messages with the index input did not exist 
thereby allowing the user 102 to determine the state of 
the conversation remotely. 

[0096] After either step 1112, 1116, 1120. 1124 or 
1126, the method arrives at step 1128. In step 1128, the 
method determines whether there are additional objects 
contained in the message for which Indices have not 
been generated. If there are additional objects, then the 
method returns to step 1 1 04 to retrieve a next object and 
generate Indices for that object. If there are not addition- 
al objects, then the method proceeds to determine 1 1 30 
the position of the objects In the message relative to 
each other. This is done in the same manner that the 
multimedia message system 110 generates an image 
for a display device directly connected to the multimedia 
message system 110. Then images of the indices 730 
are generated 1132 and they are positioned proximate 
their respective object. The indices 730 are preferably 
positioned near the upper left corner of the thumbnail 
for the object. Next, the images of the links 720 are gen- 
erated 1134. In step 1136, the method retrieves mes- 
sage header 702 and generates an image of it. The 
method also produces an image of the information 704 
for remote access including instruction, fax/phone num- 
bers, user ID numbers. Message ID numbers, reply in- 
structions, etc. This information is preferably in a ma- 
chine-readable format. The method then adds other In- 
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formation, such as a bar code or a URL to be used in a 
cover sheet when faxing new documents back into (and 
adding them to) the message. Depending on preferenc- 
es, the documents referenced in a message might also 
be automatically faxed (or one page from each), and 
thus, such additional documents would also be retrieved 
and generated in this step. Once all of the constituent 
components of the message 700 have been generated, 
they are combined into an annotated image 1138 and 
the annotated message 700 is delivered (Step 938 of 
Figure 98) to the visual output device 106. 

Remotely Retrieving Multimedia Messages 

[0097] Figures 12A and 12B show the preferred meth- 
od for retrieving visual or audio portions of a multimedia 
message in accordance with the present invention. The 
process begins by requesting a user ID number 1202, 
receiving a user's ID number 1204, and verifying 1206 
it in the same fashion to that described above in Figure 
9A. In performing the method of Figures 12A and 12B. 
it is assumed that the user 102 has obtained an image 
of the message 700. and thus knows the various objects 
or portions that form the message 700 as well as their 
indices. The method requests a message identification 
number for the message that the user 1 02 is accessing 
and retrieving. The user 102 inputs the message ID 
number using the keypad of the telephone, and the mes- 
sage ID number is received 1212 by the system lOOa. 
The method then determines 1214 if the message ID 
number is valid by comparing it to a list of valid message 
ID numbers. If the message ID number is not valid the 
method signals 1 21 6 an error and then ends. If the mes- 
sage ID number is valid, the method prompts 1218 the 
user 1 02 to input an index number corresponding to the 
portion of the multimedia message that the user 102 
wants to retrieve. The user 102 inputs the index number 
using the keypad of the telephone, and the index 
number is received 1220 by the system 100a. The sys- 
tem 100a verifies 1222 that the index number is a valid 
one and continues in step 1226. If the index number is 
not valid one, the system 1 00a signals 1 224 such to the 
user and the retrieval process ends. 
[0098] In step 1 226, the method detenmines the aaion 
to be taken (the type of delivery for the message por- 
tion). The method next determines 1228 the message 
portion corresponding to the index by retrieving an ob- 
ject based on the message identification number, and 
the object number that is part of the index. Then the sys- 
tem 100a performs 1230 the action identified in step 
1226 with the retrieved message portion. These steps 
1226, 1228 and 1230. are described below in more de- 
tail for the exemplary system. Since at least one mes- 
sage portion has been retrieved after step 1230, the 
method continues in step 1232 by prompting 1232 the 
user 1 02 for additional message portions to be retrieved. 
The system 100a determines 1234 whether the user 
1 02 wants to retrieve additional message portions. If so 
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the method returns to step 1218 to request the index 
number for the message portion from the user 102. If 
not the retrieval process is complete and ends. 
[0099] Referring now also to Figure 1 3. an exemplary 
5 method for using indices and delivering content of a mul- 
timedia message is shown. This exemplary method cor- 
responds to the case where the system 400c is config- 
ured as shown in Figure AC and portions of a message 
as shown in Figure 7 are delivered to the facsimile ma- 
10 chine 406 or the telephone 104. The method first tests 
1302 whether index number corresponds to an action 
for an audio stream. If so, the audio clip corresponding 
to the object identified in the index is retrieved 1304 and 
then it is output 1 306 over the audio interface 302 and 
15 telephone 1 04 to the user 1 02. 

[0100] If the index number does not correspond to 
sending an audio stream, the method next determines 
1308 whether the index number corresponds to the ac- 
tion of faxing the first page of a document. If the index 
20 number does correspond to faxing the first page of a 
document, the document corresponding to the object 
identified in the index number is retrieved 1310, the first 
page of the document is prepared 1312 into a facsimile 
transmission, and the facsimile transmission is output 
25 1314 over the visual output interface 314 to the facsimile 
machine 406. 

[0101] If the index number does not correspond to fax- 
ing the first page of a document, the method next deter- 
mines 1316 v/hether the index number corresponds to 

30 the action of faxing the entire document. If the index 
number does con-espond to faxing the entire document, . 
the document corresponding to the object identified in 
the index number is retrieved 1316. the entire document 
is prepared into a facsimile transmission 1320, and the 

35 facsimile transmission is output 1314 over the visual 
output interface 314 to the facsimile machine 406. It 
should be understood that alternatively, where a user is 
requesting a number of the message to be sent by fac- 
simile (either the first pages or the entire documents) 

40 then the request for multiple messages could be com- 
bined into a single facsimile as a response to the multi- 
ple requests. 

[0102] If the index number does not correspond to 
sending the entire document, the method next tests 

45 1 324 whether the index number corresponds to the ac- 
tion of sending an image at normal size. If the index 
number is for sending an image at normal size, the im- 
age corresponding to the object identified in the index 
number is retrieved 1326, the image at its normal size 

50 is prepared into a facsimile transmission 1328, and the 
facsimile trar^mission is output 1314 over the visual 
output interface 314 to the facsimile machine 406. 
[0103] If the index number does not correspond to 
sending an image at normal size, the method next tests 

55 1324 whether the index number corresponds to the ac- 
tion of sending an image zoomed (increase or de- 
creased) in size to fill the page. If the index number is 
for sending an image zoomed in size to fill the page, the 
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image corresponding to the object identified in the index 
number is retrieved 1326. The image is zoomed to 
match the standard size for a facsimile transmission, 
and the zoomed image is prepared as a facsimile trans- 
mission 1 328. The facsimile transmission is then output s 
1314 over the visual output interface 3 1 4 to the facsimile 
machine 406. 

[0104] Finally, if the index number does not corre- 
spond to sending an image zoomed in size to fill the 
page, the method signals 1336 an error because the us- 
er has input an index that does not correspond to an 
action. Alternatively, in step 1336 other ways of sending 
a message portion could be offered to the user 102. 

Remote Message Composition and Reply ^5 

[01 05] Referring now to Figure 1 4, a preferred method 
for remotely composing and replying to a multimedia 
message will be described. Before starting the method, 
it is assumed that the user 102 has remotely accessed 
the multimedia message system 1 1 0. The user 1 02 may 
also add or append objects to an existing message. In 
such a case, the user 1 02 would be prompted to identify 
the message and other verification procedures would al- 
so be performed as has been described above for re- 
trieving messages. The user 102 may also without 
prompting just indicate a document for attachment using 
the keypad of the telephone vvhile talking. The user 1 02 
at some point signals that they want to create a message 
or reply and the input is received in step 1402. A mes- so 
sage is created 1404, and the system 100a begins to 
receive audio signals from the user 1 02 that will ulti- 
mately form an audio stream. As the user 102 is provid- 
ing audio input and it is being recorded, the system 1 00a 
monitors 1406 for a signal from the user 102 that the 35 
recording is complete or that an object should be added 
to the message. For example, while recording a mes- 
sage, the user 1 02 may simply touch the keypad to enter 
numbers corresponding to objects or documents to be 
attached. The system 1 00a tests 1 408 whether user 1 02 40 
has indicated the end of recording such as by hitting the 
# key. If the user 1 02 has ended the recording, the meth- 
od adds 1430 the audio stream being recorded to the 
message, addresses 1432 the message based on input 
from the user 102, and sends 1434 the message. ^5 
[0106] While the method for addressing 1432 and 
sending 1 434 a message just described the process as 
manual one, the present invention preferably maintains 
various user profiles, and provides for the automatic de- 
livery and receipt of reply messages. The multimedia 50 
messaging system 110 is preferably enhanced to pro- 
vide the automatic delivery of messages/conversations 
appropriate to the receiver's profile. Each user 1 02 pref- 
erably has a profile defining the preferred method for 
delivery of the reply stored in database 312. For exam- 55 
pie. a default user profile would be set to respond in a 
similar manner to how the message was received. In 
that case, replies to voice messages would get delivered 
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by the multimedia messaging system 110 to the phone 
of the origlnalcalling party (the number identified by call- 
er ID or voice recognition on original message or input 
by user) while replies to multimedia messages are sent 
by email or the world wide web. It is assumed that the 
original sender has used the mechanism for communi- 
cation that they are most comfortable with and in which 
they would like the reply. Alternatively, if the user 102 is 
known to the multimedia messaging system 1 1 0, his us- 
er profile can be retrieved, and the reply message can 
be sent according to the preferences set by the user 1 02 
as the preferred method of message notification. This is 
partic'jiarly advantageous because the user 102 may 
chanf;e their profile as desired. Thus, when they are in 
the olf Ice and have full access to an interactive compu- 
ter, display, etc., they can have messages returned as 
multimedia message, and when they are out of the office 
traveling the preferences can be set for response in 
voice message format. Furthermore, the multimedia 
messaging system 110 is additionally automated by au- 
tomatic generation of user IDs for a user that is external 
to the system 110. The voice message sent to such an 
external user, also appends an audio message with in- 
formation specifying how to access the system, their us- 
er ID, a password, etc. Thereby, an external user can 
use the methods described throughout this application 
to access multimedia messages just as any remote user 
can. 

[0107] If the user 102 has not ended the recording, 
the method detenmines 1410 whether the user 102 
wants to add an object to the message. If the user does 
not want to add an object to the message, the method 
returns to step 1406 to continue monitoring for control 
input from the user 1 20. If the user 1 02 has indicated 
that an object is to be attached, the method prompts 
1414 the user 102 to indicate the object or document 
that is to be attached to the message. 
[0108] Referring now also to Figure 15, a graphical 
representation of a deliverable image or list. 1500 of ob- 
jects 1502 provided by the present invention is shown. 
The list 1 500 of objects 1 502 preferably includes a head- 
er 1504 to uniquely Identify the list 1500, a thumbnail 
image of each object/document 1502, and a unique in- 
dex 1506 for each document. Each of the objects 1502 
in the list 1500 preferably has it own unique reference 
number or index 1506 that user 102 may input from a 
remote location to identify specifically the object that is 
to be attached. Images of list 1 500 can also be provided 
to the user 1 02 in manner similar to that used to provide 
the user 102 with an image of a message (See Figure 
1 1 A and 1 1 B above). There are other methods for input- 
ting and selecting documents as will be described below 
with reference to Figures 16-18. 

[0109] Once the user 102 has been prompted 1414 
for an object reference number 1506, and the user 102 
has provided an object reference number 1506, the 
method retrieves 1416 the object or document corre- 
sponding to the object reference number 1506 input. 



15 

y. <EP 1094656A2_I_> 



# 

29 EP 1 094 656 A2 



The method then creates 1418a link 720 from the audio 
stream being created to the retrieved object or docu- 
ment. The link 720 preferably has a positional relation- 
ship to the audio stream that is reflected by the position 
of the link 720 as rendered in an image. Then the re- 
trieved object or document is added 1420 to the mes- 
sage. Next, the user 1 02 is prompted 1 422 as to whether 
there are additional objects to attach. The method de- 
termines 1424 whether the user 1 02 has decided to at- 
tach another object. If the user 1 02 has decided to attach 
another object, the method returns to step 1414 to re- 
trieve and add the object. If not, the method returns to 
step 1406. 

[0110] Referring now to Figure 16, an alternate em- 
bodiment of a method for remotely creating and sending 
a message including a visual object is shown. The proc- 
ess will be described in terms of a document or object. 
These terms are used interchangeably to refer to any 
object such as an image, a photo, a video clip or any 
other visual input. The process starts using a telephone 
104 to call the multimedia messaging system 110, the 
user 1 02 accesses 1 604 the system 1 00a. The user 1 02 
interacts with the system 100a using either voice or key 
commands to create 1 606 a new message. The user 
102 selects 1608 an object for attachment to the mes- 
sage as will be described below with reference to Figure 
1 7. It should be understood that this step 1 608 is option- 
al. A user may add only an audio object in which case 
the message contains only the audio stream or object, 
and in the case where it is a reply only the audio object 
is added to an existing message. In either case, the cre- 
ation of the object includes the generation of an icon or 
image 706a for audio objects as has been described 
above with reference to Figure 7. The system 110 then 
receives audio input from the user and creates an audio 
stream or object. The audio stream is added to the mes- 
sage and linked 1612 to the selected visual object. The 
user 102 must then address 1614 the message to an- 
other recipient in the system 1 00a. For example, the 
message could be addressed by multimedia messaging 
system 110 in response to the user 102 keying in or 
speaking a name and using voice recognition to deter- 
mine recipient. The system 100a next sends 1616 the 
message to the recipient. The reply or message com- 
position process is further automated with a final option- 
al step of automatically notifying the user about added 
object. This notification could be done by any of a 
number of conventional mechanisms such as e-mailing, 
faxing, printing or otherwise generating an image of the 
cover sheet for the message with the added object. If 
the notification method is by return voice mail, paging, 
or calling, the delivery of the notification would not in- 
clude the cover sheet but could be a voice or text mes- 
sage according to what the notification method permits. 
[0111] Figure 17 is a flow diagram of a preferred meth- 
od for selecting a visual object for attachment in a mul- 
timedia message. This selection method starts with the 
system 110 requesting that the user 102 select either 



the most recently input message/object or listen to de- 
scriptions of attachable messages. This is done by 
prompting the user 1 02 to input one of two keypad com- 
binations. The user 1 02 then inputs 1 704 a selection that 
5 is received by the system 110. The method then deter- 
mines 1706 whether the input was a selection of the 
most recently input document. If it was the method de- 
termines 1 708 the object most recently input and saved 
into the system 100a by the user 102, and retrieves the 
10 object 1710 before continuing to step 1720. If the input 
was not a selection of the most recently input document, 
the method retrieves a next attachable document from 
a list of attachable documents corresponding to the user 
102. The system 110 preferably maintains a profile for 
15 each user 1 02 that includes a list of unattached objects 
and a list of attachable objects. A single not-yet-re- 
viewed document from the list of unattached objects for 
the user 102 is retrieved in step 1714. Then the audio 
description, if any. associated with the object is retrieved 
^0 and output to the user 1 02. The system 1 1 0 then deter- 
mines if the user 102 input a signal to attach the re- 
trieved document. If the method determines the user 
102 does want to attach the retrieved document, this is 
done in step 1720. If the method determines the user 
^5 1 02 does not want to attach the retrieved document, this 
method continues to test 1722 whether there are more 
un-reviewed documents in the list. If there are more un- 
reviewed documents in the list, the method returns to 
step 1 71 4 to retrieve the next document on the list. If not 
30 the method is complete and ends. 

[01 12] While the methods fotcreating a new message 
or replying to an existing one are described above as a 
serial process, those skilled in the art will recognize that 
the steps of creating could be done incrementally, and 
35 thus before an input, such as a voice message, has 
been finished, the creation of the message and the ob- 
ject are done, and the data is added or streamed into 
them incrementally. This allows nearly simultaneous 
communication with the recipient being able to listen to 
40 an audio object nearly the same time it is being added 
to the system 110 by the sender. Also the system I00a 
has no restrictions on two different users updating or re- 
plying to the same message at the same time. Thus, 
multiple users may add objects to messages to create 
45 a type of synchronous dialog that parallels a conversa- 
tion and use of visual aids. 

Remote Audio and Video Object Input 

50 A. Remote Audio Input. 

[0113] Those skilled in the art will recognize that the 
multimedia message system 110 can be integrated to 
include voice mail functionality as has been noted 
55 above. This functionality is also preferably augmented 
with additional capabilities for automatically indexing 
and adding voice mail messages to the multimedia mes- 
sage system 1 1 0 so that they may be remotely accessed 
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using the present invention. For example, the multime- 
dia message system 110 preferably automatically as- 
signs voice mail messages received from other people 
to a conversation based on caller ID. or if using the mul- 
timedia message system 11 0 (a TouchVerse server) as 5 
the voice mail system, the server might prompt the caller 
for a message reference. Voice mail sent/delivered by 
the multimedia message system 110 (a TouchVerse 
server) preferably has prefix and/or suffix attached that 
identify the message, caller, and number to call back. io 

B. Remote Video Input . 

[0114] In those embodiments that provide a visual in- 
put capability such as described above in Figures 4C- ^5 
4E, the present invention also provides a method for in- 
putting visual objects into the multimedia messaging 
system 110. Figure 18 will now be described in the con- 
text of a system 400c such as shown in Figure 4C, how- 
ever, those skilled in the art will understand how the 
methods could be modified for the other embodiments 
that provide a greater ability to input control signals from 
the input/output communication device 112. 
[01 15] A key aspect of the present invention is provid- 
ing the functionality of the multimedia system 110 re- 
motely. Part of this functionality is the composition of 
new messages by adding documents or adding new 
documents to a message. For existing conversations, 
the method preferably adds new documents by faxing 
the new document to the multimedia messaging system 30 
110 (server) using a printout of the message as a cover 
sheet for one or more other documents. The printout of 
the message is preferably annotated by the user 1 02 
with handwriting to indicate where the newdocument(s) 
should be positioned within the existing message. For 35 
example, the user 102 may hand draw an "X" to mark 
the location of the new document in the existing mes- 
sage. The printout of the message could contain a bar 
code, a U RL, or.other machine readable information that 
can be used by the multimedia messaging system 110 "^o 
to determine the message to which the document(s) be- 
ing faxed should be added. The cover sheet either other 
one or more other documents are then faxed to the re- 
mote access system 108 using the fax machine 405. 
[01 1 6] Referring now to Figure 1 9,.a prefenred method ^5 
for processing and automatically positioning an input 
object in a message will be described- The method be- 
gins by receiving a new facsimile transmission in step 
1902- The facsimile transmission preferably comprises 
one or more-documents. For example, the facsimile 50 
transmission could be: 1) a new document of a single 
page, 2) a new document having multiple pages, or a 3) 
cover sheet as described above and one or more addi- 
tional pages. Then user's ID is determined based either 
on the received fax, caller ID, or input from the user. The 55 
user's ID number could be determined by analyzing the 
first page of the fax for any machine-readable data or 
codes and determining which user they correspond to 
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by accessing the database 312. Alternatively, before 
ending the facsimile transmission, the user could input 
their user ID number or the system could prompt the us- 
er for their ID number. Then the method determines 
1906 if the first page of the facsimile transmission is a 
cover sheet of an existing message. This can be done 
by extracting a bar code or any other machine-readable 
codes from the image or comparing the image to other 
message images to determine if the image is similar to 
an existing message. If the fax transmission is found not 
to correspond to an existing message, then the method 
proceeds to step 1918 to add the fax as an object in a 
message and store the message in the multimedia mes- 
sage system 110. The message is preferably added to 
the database 312 and references to the user or the us- 
er's list of messages, after which the method ends. 
[0117] On the other hand, if the method determines 
1906 that the first page of the facsimile transmission is 
a cover sheet of an existing message, the method pro- 
ceeds to step 1908 to compare the received image to a 
stored image of the message for differences. This can 
be done by using conventional image comparison as will 
be understood to those skilled in the art. The differences 
may indicate any handwritten annotations or other 
marks that the user 102 has hand drawn to indicate 
where the remaining pages of the fax transmission 
should be positioned in the message 700. The system 
100a then determines whether any gestures for posi- 
tioning an object, creating links, or performing other ac- 
tions have been added to the cover sheet. If not, the 
system 1 00a determines the locations automatically us- 
ing any one of a number of algorithms such as arraqged 
by time of creation, added in the message left to right in 
position, added in the message in any open space. Var- 
ious other methods for automatically determining a lo- 
cation in the message at which to add an object, as 
known to those skilled in the art, could also be used. If 
in step 1 910, the system lOOa does recognize a gesture 
for positioning the object, creating links, or performing 
other actions, the process continues by saving the loca- 
tion of the marks and any actions specified by the marks 
on the cover sheet. Then in step 1 914, the method cre- 
ates a new object from the pages of the facsimile trans- 
mission other than the cover sheet and positions the 
new object at the location determined in either step 1912 
or 1916. 

[0118] Those skilled in the art will recognize that this 
invention may also be applied to digital photocopiers 
that are able to copy images, and then send the image 
to a specified location. Thus, a copier could be used for 
the input device of the visual object. A generic cover- 
sheet could also be faxed or copied along with a docu- 
ment in a similar fashion to create a new message hav- 
ing only the one document. The generic cover sheet 
would preferably include a machine-readable user ID. 
The cover sheet is generic in the sense that it would be 
added to user's database or list of objects without a ref- 
erence to a specific message or position in a particular 
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message. 

[01 1 9] Referring now to Figure 1 8, one process for re- 
motely inputting a visual object into ihie multimedia mes- 
saging system 110 will be described. The method be- 
gins with the user 102 calling into the multimedia mes- 
saging system 110 from a fax machine 406. The call is 
received 1802, and the system 110 generates 1804 a 
greeting and requests the user's ID number. The user's 
ID is received and verified 1806 in manner that has been 
described above for other methods. The system 1 1 0 re- 
quests a title for the object from the user 102. The user 
102 may input the title by speaking the title of the doc- 
ument being faxed into the hand set or by entering num- 
bers on the keypad. The user 102 then begins the fac- 
simile transmission 1812 in a conventional manner by 
hitting the start button on the fax machine 406 and the 
fax is received 1812 by the system 110. The fax trans- 
mission is completed 1814, the system lOOa stores the 
fax as an object In a buffer corresponding to the user 
102, and uses the title as an index. Once the transmis- 
sion is complete, the call is ended 1 8i 6 and the system 
110 now includes a new object. While the present inven- 
tion has been described with reference to certain pre- 
ferred embodiments, those skilled in the art will recog- 
nize that various modifications may be provided. Varia- 
tions upon and modifications to the preferred embodi- 
ments are provided for by the present invention, which 
Is limited only by the following claims. 
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v^^ystenjior providing remote access to a multime- 
dia message having one or more objects, the sys- 
tem comprising: 35 



a multimedia messaging system having an in- 
put and an output for storing, sending and re- 
ceiving multimedia messages; 
a message retrieval unit having an input and an 
output for retrieving the message and at least 
one object, and outputting the objects accord- 
ing to a type of the object, the message retrieval 
unit coupled to the multimedia messaging sys- 
tem, the message retrieval unit also interacting 
with a user to receive control signals; and 
an object-indexing unit having an input and an 
output for translating signals input by the user 
to corresponding commands and references to 
objects that are portions of each message, the 
object- indexing unit coupled for communication 
with the multimedia message system and the 
message retrieval unit. 



The system of claim 1, further comprising a image 
generation unit responsive to the message retrieval 
unit for sending data and images from the multime- 
dia message system to a visual output device, the 
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image generation unit coupled to the multimedia 
message system, the message retrieval unit and 
the object-indexing unit. 

The system of claim 1 , further comprising a mes- 
sage composition unit having an input and an out- 
put, the message composition unit for creating and 
sending messages, the message composition unit 
coupled to the multimedia message system, the ob- 
ject indexing unit, an audio communication device, 
and a visual input device. 

Tne system of claim 1, further comprising an audio 
interface, the audio interface coupled to an audio 
communication device and to the message retrieval 
unit, the audio interface for translating signals from 
a format for the multimedia messaging system to a 
format for the audio communication device. 

The system of claim 1 , wherein the message re- 
trieval unit sends and receives data and control sig- 
nals to and from the audio interface to produce and 
send images to a visual display device. 

The system of claim 1 , further comprising a visual 
output interface, the visual output interface for com- 
munication with a visual output device, the visual 
output interface translating data and control signals 
to a format used by the visual output device, the vis- 
ual output interface coupled to the image genera- 
tion unit and the visual output device. 

The system of claim 1 , further comprising a visual 
input interface, the visual input interface for com- 
munication with a visual input device, the visual in- 
put interface translating data and control signals 
from a format used by the visual input device to the 
format used by the multimedia messaging system, 
the visual input interface coupled to the image gen- 
eration unit and the multimedia messaging system. 

The system of claim 7, wherein the visual Input in- 
terface also includes an object input unit for receiv- 
ing objects and storing the objects in the multimedia 
message system. 

^ A method Vor remotely accessing and delivering a 
iTi uU ini ed ia message in visual form, the multimedia 
message including an object, the method compris- 
ing the steps of: 

determining whether the user has requested 

the message in visual form; 

in response to user input, generating an image 

of the message including an index for remotely 

retrieving the object included in the message; 

and 

sending the image of the message to the user. 
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10. The method of claim 9. wherein the step of sending 
the image of the message to the user comprises 
printing the image on paper. 

11. The method of claim 9. wherein the step of sending 
the image of the message to the user comprises 
sending a facsimile transmission including the im- 
age to a telephone number. 

12. The method of claim 9, further comprising the step 
of establishing communication that includes the 
steps of: 

receiving a user ID number; 
verifying the user ID number; 
requesting and receiving a password; and 
verifying the password; 

13. The method of claim 9, further comprising the step 
of 

identifying a message with a message identi- 
fication number; and 

wherein the step of determining whether the 
user has requested the message in visual form in- 
cludes receiving the messages identification 
number from the user, and comparing the message 
identification number to a list of messages identifi- 
cation numbercorresponding to messages In a mul- 
timedia message system. 

14. The method of claim 9 wherein the step of deter- 
mining whether the user has requested the mes- 
sage In visual form includes: 

prompting the user whether they want the mes- 
sage in visual form; 

receiving an input from the user indicating a 
next un-revlewed message In visual form is 

wanted; and 

selecting a next message in a list of un-re- 
viewed messages. 

15. The method of claim 9, further comprising the step 
of determining a mechanism for delivering the mes- 
sage in visual form. 

16. The method of claim 15, wherein the step of deter- 
mining a mechanism for delivering the message fur- 
ther comprises the steps of: 

determining whether the user has access to a 
fax machine; 

retrieving a fax number from the user and con- 
figuring a multimedia message system to send 
the image as a facsimile to the number re- 
trieved If user has access to a fax machine 
determining whether the user has access to e- 
mail; 
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retrieving an e-mail address from the user and 
configuring the multimedia message system to 
send the Image as an e-mail to the address re- 
trieved If user has access to e-mail; 
5 determining whether the user has access to a 

printer; 

retrieving a printer name from the user and con- 
figuring the multimedia message system to 
print the image on the retrieved printer if user 
10 has access to the printer; 

determining whether the user has access to a 
display device; and 

configuring the multimedia message system to 
send the image on the display device if user has 
15 access to the display device. 

17. The method of claim 9, wherein the step of gener- 
ating an image of the message further comprises 
the steps of: 

20 

determining the objects in the message; 
generate an image for an object In the message 
determining position of the object in the image 
of the message; 
^5 generating or retrieving an index for the object; 

generate an image of the index and positioning 
it proximate the image of the object; and 
producing an image of the message by combin- 
ing the image of the object and the image of the 
30 index. 

18. The method of claim 17, further comprising the step 
of generating links between objects, and wherein 
the step of producing the image includes using the 

35 links. 

19. The method of claim 1 7, wherein the step of gener- 
ating an index produces an index based upon a type 
of the object, methods of delivery possible for that 

40 type of object, and an object number that is unique 
on a message basis. 

20. The method of claim 1 7, wherein the step of gener- 
ating an index produces an index based upon a user 
identification number, type of the object, and a 
number of the object created from an author. 

21. A method for remotely retrieving an object portion 
of a multimedia message, the method comprising 

50 the steps of: 

receiving a request from a user for an object 
portion of a multimedia message; 
receiving an index to the object portion; 
55 determining and retrieving the object portion to 

which the index corresponds; and 
performing an action with the retrieved object 
portion corresponding to the Index received. 
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22. The method of claim 21 further comprising the step 
of sending a visual image of the message to the us- 
er prior to the step of receiving the request from the 
user for an object portion. 

5 

23. The method for remotely retrieving an object portion 
of claim 18. the method further comprising the steps 
of: 

receiving a user identification number; io 
verifying the user Identification number; and 

wherein the step of receiving a request from 
a user for the object portion of the multimedia mes- 
sage Includes the steps of receiving a message is 
Identification number and verifying the message 
Identification number. 

24. The method for remotely retrieving an object portion 
of claim 1 8. the method further comprising the steps 
of: 



sending the first page of the object, 

sending all pages of the object, sending the 
object at normal size, or 
sending the object zoomed to fit a prede- 
fined page size; 

modifying the object to generate an image ac- 
cording to the step of determining whether the 
index corresponds to sending the first page of 
the object; and 
sending the modified image. 

28. The method for remotely retrieving an object portion 
of claim 18, the method further comprising the steps 
of: 

determining whether the index corresponds to 

outputling a printed document; and 

if the index corresponds to outputting a printed 

document: 



determining whether the Index corresponds to 
outputting audio; and 

If the index corresponds to outputting audio: 

retrieving an audio object corresponding 
for the message; and 
outputting the audio object to the user via 
an audio communication device. 

25. The method for remotely retrieving an object portion 
of .claim 1 8. the method further comprising the steps 
of: 

determining whether the index corresponds to 
outputting a facsimile; and 
if the index corresponds to outputting a facsim- 
ile: 

retrieving an object corresponding to the 
index; 

preparing a facsimile; and 

outputting the facsimile to the user via a 

visual communication device. 

26. The method of claim 25, wherein the user has input 
multiple indices to request a plurality of object por- 
tions, and wherein the step of outputting the facsim- 
ile to the user via a visual communication device 
provides the plurality of object portions in a single 
facsimile transmission. 

27. The method for remotely retrieving an object portion 
of claim 21 , the method further comprising the steps 
of: 

determining whether the Index corresponds to 



25 



30 



35 



40 



45 



50 



55 



retrieving an object corresponding to the 
Index; 

preparing a file for printing; and 
outputting the file to the user via a printer 
Identified by the user 

A method^for remotely creating a multimedia mes- 
g^age-including an object, the method comprising 
the steps of; 

receiving input from a user to compose a mes- 
sage; 

creating a message; and 

receiving audio input, creating an audio 
object, and adding the audio object to the mes- 
sage, said audio object including a visually dis- 
playable portion.; 

30. The method ofclaim 29 further comprising the steps 
of: 

monitoring for a signal to Include a visual object 
in the message; 

receiving an index to the visual object to be in- 
cluded; 

adding the visual object to the message; and 
creating a link between the audio object and the 
visual object; 

31 . The method of claim 29 further comprising the step 
of addressing the message based upon user input. 

32. The method of claim 29 further comprising the step 
of automatically delivering the message using user 
profiles. 
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33. The method of claim 29further comprising the step 
of automatically delivering using a format matching 
a delivery of an original message to which the mes- 
sage relies. 

5 

34. A method for remotely creating and sending a mes- 
sage including a visual object, the method compris- 
ing the steps of: 

accessing a multimedia messaging system; io 
creating and storing a new message; 
selecting an object for inclusion in the mes- 
sage; 

receiving audio input from the userand creating 
an audio object; ^5 
adding the audio object to the message; 
linking the audio object to the selected visual 
object; and 

adding the selected visual object to the mes- 
sage. 

35. The method of claim 34, further comprising the 
steps of: 

addressing the new message to a recipient; 25 
and 

sending the message to the recipient. 

36- The method of claim 34, wherein the step of select- 
ing an object for inclusion in the message, further 3o 
comprises the steps of: 

determining whether to include most recently 

input objects; 

determining object most recently Input to the 35 
multimedia messaging system by the user; 
retrieving object determined to be most recently 
input; and 

adding the retrieved object to the message. 

40 

37. The method of claim 34, wherein the step of select- 
ing an object for inclusion in the message, further 
comprises the steps of: 

determining whether to listen to description of 
recently input objects; 

retrieving a next attachable object from a list of 
attachable objects; 

retrieving and outputting an audio clip associ- 
ated with the next attachable object; so 
determining whether a signal was received to 
attach the next attachable object; and 
attaching the next attachable object if the signal 
was received to attach. 

38. (a metho^or remotely inputting an object into a mul- 

tiitiedia messaging system, the method comprising 
the steps of: 



initializing communication with the multimedia 

messaging system; 

receiving information from a user; 

storing the received information in an object; 

storing the object in the multimedia messaging 

system. 

39. The method of claim 38 wherein the object is an au- 
dio object, and the step of providing a name for the 
object includes automatically assigning objects re- 
ceived to a message based on the identity of the 
user. 

40. The method of claim 38 wherein the step of receiv- 
ing Information from a user includes receiving a fac- 
simile transmission having a first page and addition- 
al pages, and wherein the method further compris- 
es the steps of: 

determining whether the first page of the fac- 
simile transmission corresponds to an existing 
message; 

adding the object to the existing message, If it 
Is determined that the first page of the facsimile 
transmission corresponds to the existing mes- 
sage; and 

adding the object to a message referenced to 
the user If the first page of the facsimile trans- 
mission does not correspond to an existing 

message. 

41. The method ofxiaim 40 wherein the step of adding 
the additional pages as an object to the existing 
message further comprises the step of positioning 
the object in the message using an automatic layout 
routine based on open space or time of addition. 

42. The method of claim 40 wherein the step of adding 
the additional pages as an object to the existing 
message further comprises the steps of: 

comparing the first page to an image of an ex- 
isting message; 

determining whether there is an annotation on 

the first page; 

determining a location of the annotation; and 
adding the object to the messages at a position 
corresponding to the location of the annotation. 

43. The method of claim 38 wherein the step of receiv- 
ing information from a user Includes receiving an 
audio signal, and wherein the method further com- 
prises the steps of: 

determining whether the audio signal corre- 
sponds to an existing message; 
adding the object to the existing message, if it 
Is determined that the audio signal corresponds 
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to an existing message; and 
adding the object to a message referenced to 
the user if it is detenmined that the audio signal 
does not correspond to an existing message. 

44. The method of claim 38 wherein the receiving infor- 
mation from a user includes receiving a facsimile 
transmission of an image of a message having 
handwritten annotations, and the step of storing the 
retrieved information comprises translating the 
handwritten annotations, and copying an existing 
object identified by the handwritten annotations. 

45. The method of claim 38 wherein the step of initial- 
izing communication comprises the steps of: 

calling into the multimedia messaging system; 
and 

requesting and receiving a user's Identification 
number. 

46. The method of claim 38 wherein the step of receiv- 
ing information from a user comprises receiving a 
facsimile transmission. 

47. The method of claim 38 wherein the step of provid- 
ing a name for the object comprises receiving audio 

lis from user 

48. V A method>for automatically generating and sending 
aTr-ev0fview visual representation of a message 

stored in a multimedia messaging system, the 
method comprising the steps of: 

receiving a triggering event; 
identifying a user from the triggering event; 
identifying messages corresponding to the 
identified user; ^ 

generates an overview visual representatio n 
formed of a representation for each message 
identified and an index for each representation; 
and 

sending the overview visual representation. 

49. The method of claim 48 wherein the overview visual 
representation includes visual representations of 
audio objects. 

50. The method of claim 48 wherein the triggering event 
is a user call into the multimedia messaging system 
or an addition of a message addressed to the user. 
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51 . The method of claim 48 wherein the triggering event 
is a user call into the multimedia messaging system, 
and method includes the steps of: 55 

identifying the user with caller ID; 
retrieving a profile for the user; 



determining a mechanism for delivery and a lo- 
cation for delivery of the overview visual repre- 
sentation using the retrieved profile: and 

wherein the step of sending, sends the over- 
view visual representation to the location deter- 
mined using the delivery mechanism determined. 

52. A method for remotely creating and sending an ob- 
ject, the method comprising the steps of: 

accessing a multimedia messaging system; 
creating and storing a new message or Identi- 
fying an existing message: 
creating an audio object; 
receiving audio input from a first user and stor- 
ing it in the audio object; 
adding the audio object to the message; and 
positioning the object in the message using an 
automatic layout routine based on open space 
or time of addition of the object. 

53. The method of claim 52 further comprising the steps 
of: 

identifying recipient for the message; and 
notifying the identified recipients once the au- 
dio object is created- 

54. The method of claim 52 wherein a second user can 
access the audio object while the first user is pro- 
viding the audio input. 

55. The method of claim 52 wherein one or more other 
users can retrieve the message simultaneously. 

56. The method of claim 52 wherein one or more other 
users can reply to the message simultaneously. 

57. A method for automatically adding users to a multi- 
media messaging system, the method comprising 
the steps of: 

accessing the multimedia messaging system 
by an existing user; 

receiving identification and contact information 
for a new user from the existing user; 
generating an identification number for the new 
user in the multimedia messaging system; 
generating a password for the new user; 
generating a notification message to the new 
user, the notification including instructions on 
how to access the multimedia system, the gen- 
erated identification number, and the generated 
password; and 

sending the notification message to the new us- 
er. 
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58. The method of claim 57. wherein the step of receiv- 
ing identification information for the new user is per- 
formed by receiving a caller ID number, and the new 
user identification number is based on the caller ID 
number. 

59. The method of claim 57, wherein the step of gener- 
ating a notification message Is generating an audio 
message, and the step of sending is calling the new 
user and providing the audio message. 
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